Skip to content

[DRAFT] FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader#2175

Draft
3nesdeniz wants to merge 3 commits into
microsoft:mainfrom
3nesdeniz:feat/agentic-prompt-injection-boundary-dataset
Draft

[DRAFT] FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader#2175
3nesdeniz wants to merge 3 commits into
microsoft:mainfrom
3nesdeniz:feat/agentic-prompt-injection-boundary-dataset

Conversation

@3nesdeniz

@3nesdeniz 3nesdeniz commented Jul 13, 2026

Copy link
Copy Markdown

Description

Closes #2174.

This PR adds a remote loader for the public CC BY 4.0 Agentic Prompt-Injection Boundary Pairs dataset:

  • 1,200 English records organized as 600 controlled benign/attack pairs;
  • 50 scenario-isolated enterprise and agentic workflows;
  • 12 prompt-injection and trust-boundary attack families;
  • train, validation, test, or combined loading;
  • attack-only default plus benign and full-pair modes;
  • typed filters for label, split, attack family, and source context;
  • pair, scenario, source, boundary, expected-action, version, and revision provenance on each seed.

The loader maps each row to a literal SeedPrompt and pins the immutable v1.0.0 Hugging Face revision. Before filtering, it validates required fields, label-dependent values, allowed families and source contexts, split metadata, duplicate IDs, and complete pair integrity.

Family filtering uses pair_family so full-pair mode preserves both sides of a controlled pair. Agent-security decision labels and attack families remain in metadata rather than being misclassified as PyRIT harm categories. Pair members remain independent evaluation cases; pair_id preserves the relationship without collapsing two prompts into one SeedGroup.

Source and release pipeline: https://github.com/3nesdeniz/agentic-prompt-injection-boundary-pairs

I am the dataset author and maintain the tagged source release. The dataset is synthetic, manually reviewed, and contains no customer or private data.

Tests and documentation

  • Focused loader and provider validation: 44 passed
  • Full dataset unit-test package: 1,171 passed
  • Full unit suite after hardening: 10,637 passed, 119 skipped
  • Targeted pre-commit checks passed, including Ruff, changed-file type checking, notebook sanitation, documentation validation, secret detection, and JSON checks
  • Strict Jupytext validation passed
  • Live pinned-release loading confirmed 600 default attack seeds and 240 test rows forming 120 complete pairs
  • The public release was verified as 1,200 rows, 600 pairs, 50 scenarios, with matching JSONL/Parquet artifacts and checksums
  • One real matched pair from each of the 12 attack families was manually inspected

The loader is registered in dataset discovery and documented in the loading guide, notebook, and project bibliography.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader

1 participant